Repository navigation
feat(gateway): integrate the Wave 4 gateway features - #53
Merged
ALikesToCode merged 121 commits intoOct 9, 2026
Merged
Conversation
Validate non-streaming managed results against a caller-supplied JSON schema behind OUTPUT_SCHEMA_VALIDATION_ENABLED (default off). Invalid results return 502 output_schema_violation with bounded, content-free paths; no regeneration or repair.
Carry reviewed per-protocol fields outside the Chat pivot behind PROTOCOL_EXTRAS_ENABLED (default off), restore them only for providers that admit each field, and report retained versus re-emitted fields without values.
Add multillm.tools.discover on both MCP endpoints behind DEFERRED_TOOLS_ENABLED (default off): grant-filtered local ranking with complete schemas, signed short-lived cursors, and grant and argument rechecks before every tool call. Grants live in the new tool_grants table (migration 0018) with compatibility rows for current tools.
…o feat/deferred-tools # Conflicts: # package.json
Share complete, deterministic non-streaming responses across Flask Containers and native Worker routes behind GENERATION_CACHE_SHARED_ENABLED with GENERATION_CACHE_BACKEND=d1-r2 (default memory, unchanged). Entries are isolated by principal and resolved policy, stored as D1 metadata (migration 0019) with bounded bodies in the existing R2 bucket, and zero-content retention bypasses storage.
# Conflicts: # package.json # worker/container-env.mjs # worker/d1-schema.mjs
Track each budgeted request through reserve, dispatched, settled or unknown, and reconciled states behind USAGE_RESERVATIONS_ENABLED (default off). An unknown completion keeps a conservative estimated hold instead of a zero charge or a timed refund; USAGE_HOLD_REVIEW_AFTER_SECONDS marks old holds for review, and admins reconcile with measured evidence or an audited adjustment. Reservations are stored in D1 (migration 0020).
# Conflicts: # package.json # worker/container-env.mjs # worker/d1-schema.mjs
Bind an authenticated session and role lane (main, delegation or aux) to its approved model tier behind SESSION_TIER_MODE=sticky (default off), keep the tier through unresolved tool calls, and change it only at a safe user-turn boundary under existing policy. Bindings are hashed and stored in D1 (migration 0021) and roleplay Durable Object storage; expiry, revision or credential failure clears them.
# Conflicts: # package.json # worker/container-env.mjs # worker/d1-schema.mjs
Prefer the credential that actually completed a reusable prefix on authenticated managed automatic routes behind PROMPT_CACHE_AFFINITY_ENABLED (default off). Bindings come only from classified successful attempts, are hash-only and bounded by TTL and LRU, and never override cooldown, grants, budgets or fallback eligibility.
Carry one monotonic budget from X-MultiLLM-Deadline-Ms through setup, admission, retry waits, provider calls, preflight and stream reads. Expiry returns 504 before commitment or a protocol error event after it, cancels the upstream once and settles usage as unknown. Requests without the header are unchanged.
Evaluate operator alert rules over content-free aggregates (spend estimate, unknown-price coverage, open circuits, exhausted pools, consecutive provider failures) behind GATEWAY_ALERTS_ENABLED (default off). Rules and delivery events live in D1 (migration 0022). The Worker's scheduled run delivers deduplicated webhooks to allowlisted HTTPS origins (GATEWAY_ALERT_WEBHOOK_ALLOWLIST), at most three attempts per event. Administrators manage rules at /admin/alerts, which returns 404 while the flag is off.
Accept Idempotency-Key on managed nonstreaming chat behind MANAGED_IDEMPOTENCY_ENABLED (default off). An atomic D1 claim (migration 0023) precedes admission and dispatch, binds the key to the principal, path, normalized body and routing policy, and records the provider handoff durably. Concurrent duplicates get 409 request_in_progress, a changed request under the same key gets 422, and a completed response replays from R2 with X-MultiLLM-Idempotency: replayed and no new provider call. With the flag off, the intelligence gateway keeps rejecting the header and raw provider routes are unchanged.
Behind BANDIT_MODE (off, shadow or recommendation; default off), build task/model cells from stored evaluation results and nonce-protected administrator feedback, gate them on sample count, decay and the evaluation noise floor, and propose at most a one-position move in an automatic route's eligible order. Neither mode calls a provider or writes route policy; a proposal is applied only through the reviewed configuration snapshot apply with its revision check. Unknown cost or quality never ranks as favourable.
Behind CONTEXT_PAGING_ENABLED (default off), a managed client that declares the multillm_context_retrieve capability can have complete older exchanges replaced by signed retrieval handles instead of being summarized or truncated. Pages keep the exact stored JSON, never split a tool call from its results, and are scoped to the principal, session and policy revision. Bodies go in the existing R2 bucket and metadata in D1 (migration 0024), with 64 KiB per page, 1 MiB per session and a one-hour TTL. Zero-content retention disables paging, and a request whose protected content still does not fit returns 413.
Register ollama, vllm, lmstudio, llamacpp and sglang as OpenAI-compatible providers only when their base URL variable is set (all unset by default). URLs are validated to an explicit /v1 endpoint, plain HTTP is limited to loopback and private addresses, and an optional provider key is used when configured; an explicitly configured local server may be called without one. No port scanning, discovery or downloads; catalog entries carry declared capabilities with provenance.
Send content-free request observations to explicitly configured Langfuse and Helicone collectors when OBSERVABILITY_EXPORTERS_JSON lists them. Delivery uses a bounded in-memory queue with retries limited to transport failures, 429 and 5xx, and counts acknowledgements per exporter. Credentials come from variables ending in _EXPORT_CREDENTIAL. With the setting unset, nothing is sent and OTLP export is unchanged.
…o feat/observability-exports # Conflicts: # package.json
…o feat/prompt-injection # Conflicts: # package.json
… integrate/gateway-waves-1-5 # Conflicts: # package.json
…grate/gateway-waves-1-5 # Conflicts: # package.json
On a slow runner the 20 ms deadline expired before the request reached the fetcher, so the hold was released instead of kept as unknown. The fetcher never settles, so a 1000 ms deadline still fires after handoff.
…ails An intermittent CI failure returned 502 with no log, so the assertion now reports the provider call count and the response body.
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
ALikesToCode
merged commit Oct 9, 2026
bf939e0
into
integrate/gateway-waves-1-3
5 of 7 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch merges all ten Wave 4 feature branches on top of the Wave 3 integration branch (#39) and wires them together. It is based on
integrate/gateway-waves-1-3, so this diff shows only Wave 4.OUTPUT_SCHEMA_VALIDATION_ENABLEDPROTOCOL_EXTRAS_ENABLEDDEFERRED_TOOLS_ENABLED, migration 0018PROMPT_CACHE_AFFINITY_ENABLEDX-MultiLLM-Deadline-Ms, capped byGENERATION_DEADLINE_MAX_MSGENERATION_CACHE_SHARED_ENABLEDwithGENERATION_CACHE_BACKEND=d1-r2, migration 0019USAGE_RESERVATIONS_ENABLED, migration 0020SESSION_TIER_MODE=sticky, migration 0021GATEWAY_ALERTS_ENABLED, migration 0022MANAGED_IDEMPOTENCY_ENABLED, migration 0023The six migration PRs form one stack (#38 → #43 → #44 → #45 → #46 → #47) so each keeps its numbering contiguous. All ten are merged here.
Integration changes
Flask
The single registrar
register_gateway_extensionsnow registers the post-authentication hooks in this fixed order. It is guarded so a second call does nothing.before_requestguard, unchanged./v1/responses only.The managed pipeline lives in the new
services/managed_turn.py. It runs for managed Chat Completions, translated protocol requests and native protocol requests:_multillm. Neither field reaches a provider.session_tier, then pick candidates from one list. Eligibility, the approved tier and the prompt cache affinity order are applied to that same list.mark_dispatched()runs right before submission, after the cancellation checks, and again for lazily started streams. Provider timeouts are bounded by the deadline.routes/unified.pystays under 1,000 lines; the new logic is inmanaged_turn.py.Storage plumbing:
/v1/state/alerts(control-state D1 domain) and reservations at/v1/reservations.D1ReservationStoreand never fall back to SQLite.Worker
/v1/reservationsis mounted on the private outbound handler. The control-state dispatcher gains analertsdomain, which returns 404 while alerts are off.worker/scheduled-maintenance.mjs. With both off it registers no extra work. It handles at most three shared-cache cleanup batches, while that cache is on, and alert delivery, while alerts are on. Spend and unknown-price alerts are aggregated fromusage_daily. Circuit, pool and provider-failure alerts come from Flask's existing passive health observations, and no new probes are added.Coordinator fix: idempotency contract check
The idempotency hook checks intelligence requests against the strict chat contract before admission. It used to run that check on the raw body. As a result, with idempotency on, an
/intelligenceorauto:intelligencerequest that carried anIdempotency-Keyand an enabled gateway option was rejected with 400invalid_routing_request. Affected options:multillm_output_validation,_multillmorsession_tier.The managed pipeline removes those options before its own parse. The hook now checks the same view (
managed_turn.contract_payload).tests/test_managed_idempotency.pycovers the case; the test fails without the fix.Coordinator fix: forwarded deadlines
The Worker's forwarding step used to replace the caller's
X-MultiLLM-Deadline-Mswith only the internal remaining-budget header. Flask ignores that internal header until a transport verifier exists. So a forwarded request, which is every Flask-served request in production, ended up with no deadline at all in Flask.The Worker now keeps the public header and adds the internal one. Flask applies the smaller of the two, and today that is the public header alone. The doc and tests (
tests/test_wave4_worker.mjs) are updated.Behaviour change with all flags off
On forwarded requests, the Worker strips caller copies of
X-MultiLLM-Internal-Deadline-Msbefore calling the Container. Before, a client could send them. Flask ignores that header in any case (see below), so this only removes spoofable input. The public deadline header passes through as before.Known limits
X-MultiLLM-Deadline-Mson Flask's clock, so time spent in the Worker is not counted.auto:/cascade:models. Explicitprovider:modelrequests are not claimed.routes/free_routes.pydoes not pass through the managed accounting and deadline boundary.usageMetadata, so those holds stay open until reconciled.session_tiersD1 table has no reader yet; a D1 adapter is a follow-up.Before merging to main
Six D1 migrations are included:
0018_tool_grants.sqlthrough0023_idempotency.sql. A push to main does not apply them. Apply them first:Tests
tests/test_wave4_registrar.py: hook order and storage plumbing;tests/test_wave4_managed.py: the managed pipeline through the registered routes;tests/test_wave4_worker.mjs: native order, private routes and scheduled maintenance.npm run test:worker: 1,486 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 23 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.Wave 5 feature PRs (#48–#52) target this branch.